Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed load_samples to wrap variables in quotes to prevent YAML parsing errors #5858

Merged
merged 1 commit into from
Mar 8, 2025

Conversation

NevilleS
Copy link
Contributor

@NevilleS NevilleS commented Mar 8, 2025

Description Of Changes

Unfortunately the variable expansion logic in the FIDES__DATABASE__LOAD_SAMPLES handling can result in loading a character like &, which YAML parsing then tries to interpret as an anchor, and then it fails to load the sample data.

Fides' startup routine is resilient to that failure, it just continues starting up but aborts loading the rest of the sample data, which is a subtle footgun and caught us in a test environment recently!

This addresses this by wrapping all the variables we load from the template connections in quotes, and adds some tests to exercise that.

Code Changes

  • Update sample connections YML to wrap variables in quotes
  • Add test to ensure that sample connections YML loads safely with special characters in variables
  • Add test to demonstrate what will/won't fail to load, for future engineers

Steps to Confirm

  1. Run unit tests and confirm
  2. Use FIDES__DATABASE__LOAD_SAMPLES=1 and insert ENV variables like FIDES_DEPLOY__CONNECTORS__POSTGRES__HOST=&123 and confirm that YAML errors do not occur

Pre-Merge Checklist

  • Issue requirements met
  • All CI pipelines succeeded
  • CHANGELOG.md updated
    • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
    • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Followup issues:
    • Followup issues created (include link)
    • No followup issues
  • Database migrations:
    • Ensure that your downrev is up to date with the latest revision on main
    • Ensure that your downgrade() migration is correct and works
      • If a downgrade migration is not possible for this change, please call this out in the PR description!
    • No migrations
  • Documentation:
    • Documentation complete, PR opened in fidesdocs
    • Documentation issue created in fidesdocs
    • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
    • No documentation updates required

Copy link

vercel bot commented Mar 8, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
fides-privacy-center ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 8, 2025 0:35am
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Mar 8, 2025 0:35am

Copy link

codecov bot commented Mar 8, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.01%. Comparing base (7f21d2e) to head (204aa7b).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5858   +/-   ##
=======================================
  Coverage   87.01%   87.01%           
=======================================
  Files         407      407           
  Lines       25123    25123           
  Branches     2706     2706           
=======================================
  Hits        21862    21862           
  Misses       2678     2678           
  Partials      583      583           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@adamsachs adamsachs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 thanks for the good test coverage!

@NevilleS NevilleS merged commit 238bad5 into main Mar 8, 2025
38 checks passed
@NevilleS NevilleS deleted the ns-fix-load-samples-quote-vars branch March 8, 2025 02:39
Copy link

cypress bot commented Mar 8, 2025

fides    Run #12644

Run Properties:  status check passed Passed #12644  •  git commit 238bad5aaa: Fixed load_samples to wrap variables in quotes to prevent YAML parsing errors (#...
Project fides
Branch Review main
Run status status check passed Passed #12644
Run duration 00m 50s
Commit git commit 238bad5aaa: Fixed load_samples to wrap variables in quotes to prevent YAML parsing errors (#...
Committer Neville Samuell
View all properties for this run ↗︎

Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 5
View all changes introduced in this branch ↗︎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants